Ask model to try again if it produced a response without text or tool call parts (e.g. only thinking) #2469
Conversation
|
@ethanabrooks Thanks, which models have you seen output only thinking with no text or tool calls to follow it? |
ethanabrooks
force-pushed
the
eb/handle-thinking-only-requests
branch
4 times, most recently
from
7247ccd to
6139ac7
Compare
ethanabrooks
force-pushed
the
eb/handle-thinking-only-requests
branch
2 times, most recently
from
269ef33 to
e40e7a5
Compare
ethanabrooks
force-pushed
the
eb/handle-thinking-only-requests
branch
from
e40e7a5 to
a06e745
Compare
There was a problem hiding this comment.
@ethanabrooks Thanks Ethan, this fix makes sense. Can you please add a test with a simulated thinking-only response using FunctionModel?
| @@ -482,6 +499,12 @@ async def _run_stream() -> AsyncIterator[_messages.HandleResponseEvent]: | |||
| self._next_node = await self._handle_text_response(ctx, last_texts) | |||
| return | |||
|
|
|||
| # If there are no preceding model responses, we prompt the model to try again and provide actionable output. | |||
| breakpoint() | |||
There was a problem hiding this comment.
Please remove this breakpoint
| if thinking_parts: | ||
| # Create the retry request using UserPromptPart for API compatibility | ||
| # We'll use a special content marker to detect this is a thinking retry | ||
| retry_part = _messages.UserPromptPart( |
There was a problem hiding this comment.
Let's use a RetryPromptPart as we do here:
pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py
Lines 539 to 541 in 07b8a22
| # Create the retry request using UserPromptPart for API compatibility | ||
| # We'll use a special content marker to detect this is a thinking retry | ||
| retry_part = _messages.UserPromptPart( | ||
| 'Based on your thinking above, you MUST now provide ' |
There was a problem hiding this comment.
Since we don't currently send back the thinking parts, the model won't know what this refers to. Can we simplify the message to be something more generic like "Responses without text or tool calls are not permitted." and see if that's enough to get the model to do better?
Note that wrapping it in a RetryPromptPart as suggested above already adds Fix the errors and try again. after this message:
pydantic-ai/pydantic_ai_slim/pydantic_ai/messages.py
Lines 646 to 656 in 07b8a22
| actionable output alongside their thinking content. | ||
| """ | ||
| thinking_parts = [p for p in parts if isinstance(p, _messages.ThinkingPart)] | ||
| if thinking_parts: |
There was a problem hiding this comment.
I think we can drop this check and always try to get the model to try again instead of raising a hard error.
DouweM
changed the title
UnexpectedModelBehavior)
Fixes #2480